The invention generally relates to technology that facilitates user and time-based content interaction. In particular, the invention relates to systems and methods that facilitate such interactions by determining, optionally without relying on watermark matching, whether a match exits between user sampling data with time-based content.
Traditionally, electronically transmitted time-based content involved a one-way rather than an interactive relationship between a source of content, e.g., a television station or radio broadcaster, and an audience. Although call-in television and radio programs allowed for a limited degree of real-time interactivity, such programs were the exception rather than the rule. To obtain information relating to audience behavior, sources of time-based content often relied on third-party services and devices such as those provided by Arbitron, Inc. (Columbia, Md.), Such services and devices generally serve to identify and measure an audience's exposure to time-based content and to provide information useful to original sources of time-based content, other sources of content distribution, advertisers, and the like.
With the emergence of multiple, overlapping media distribution pathways, as well as the wide variety of available user systems (e.g. personal, notebook and tablet computers, personal digital assistants, smart phones, Internet appliances, televisions, radios, etc.) for receiving time-based content, the potential of interactivity with an audience has grown exponentially. From the perspective of media distributers, interactivity gives rise to commercial opportunities that were previously impossible. Numerous commercial entities, e.g., Arbitron, Inc. (Columbia, Md.), Audible Magic Corporation (Los Gatos, Calif.), and Shazam Entertainment Ltd. (London, United Kingdom), have sought to capitalize on such potential for interactivity.
From an identification and measurement perspective, there are a number of approaches relevant to user and content interactivity. A first approach involves watermarking, i.e., encoding time-based content with program identification data or other ancillary code. For example, U.S. Patent No. RE 42,627 to Neuhauser et al. discloses watermarking technology. A second approach involves extraction and subsequent pattern matching of “signatures” of the time-based content. U.S. Pat. No. 8,086,445 to Wold et al. provides an example of such signature matching technology. U.S. Pat. No. 7,483,835 to Neuhauser et al. sets froth technology that employs aspects of both watermarking and signature matching. Such approaches may function to assess audience ratings, detect copyright piracy, calculate royalties and verify airings.
From an audience standpoint, consumer electronic devices such as smart phones are now available with the capability to recognize audio/visual based content using technology powered by computer vision software/embedded-hardware. For example, smart phones equipped with optical technologies now allow smart phone to read one- or two-dimensional barcodes on products and print materials. Once a match is found, the smart-phone user may be directed to a website and provided with further information pertaining to the products or print materials.
Bar-code identification and similar technologies for recognizing static rather than time-based content are adequate to provide user-content interactivity in limited circumstances. Typically, such technologies are useful in providing user interactivity with content appearing in print media such as magazines and newspapers. However, such technologies fall short when users wish to interact with time-based content such as radio and television.
Commercial entities such as Shazam Entertainment Ltd. (London, United Kingdom) have attempted to address user desires to identify with time-based content. For example, Shazam Entertainment distributes downloadable apps allow a user to sample music from a primary device, e.g., a radio or television, by using a secondary device, e.g., a mobile phone having a built-in microphone. An acoustic fingerprint or signature based on the sample is created, and is compared against a central database for a match. If a match is found, information such as the artist, song title, and album, is relayed back to the user. Exemplary technology associated with Shazam Entertainment are set forth in U.S. Patent Application Publication Nos. 20120079515, 20120076310, 20110273455, 20110099197, 20080082510, 20020161741, and 20020072982, as well as in U.S. Pat. Nos. 8,190,435, 8,086,171, and 7,881,657.
However, known technologies such as that currently provided by Shazam Entertainment fail to address a number of problems associated with interactions between users and time-based content. For example, a user may wish to interact with 15-30 second commercial spot or other content segment of a similar duration, e.g., a pre- or post-spot announcement occurring during a radio or television program. With previously known technologies, the user would have a small window of time to sample the content segment. By the time the user decides that he or she would like to identify and/or interact with the content segment, and pulls out his or her cellphone to sample the content segment, the segment may be substantially or completely over. That is, there may not be sufficient time to obtain an adequate sample of the content segment to identify the content segment.
Another problem associated with previously known signature-based technologies is that they typically require pre-indexing of time-based content before the content is transmitted to the primary device. That is, only pre-indexed content can be identified later. This limitation does not generally pose a problem for static content such as printed material or a problem for identifying prerecorded, time-based content, such as songs broadcast via radio since such content is created long before they are distributed publicly. However, this limitation does pose a problem for other types of content, e.g., live broadcasts or content produced by those who do not have timely access to pre-indexing technology.
A further problem associated with prior art technologies is the speed of their performance. For example, in prior art technologies for which it is necessary to determine whether there is a match between sampled data and indexed time-based content in a database, the speed at which such a determination may be carried out is generally inversely proportional to the number of entries in the database. As more time-based content is created over time, massive databases with enormous quantities of indexed time-based entries must be employed to increase the likelihood of finding a match between sampled data and content. In turn, throughput or matching speed may be compromised.
Accordingly, opportunities exist to overcome disadvantages associated with known technologies. Such disadvantages may be overcome by employing systems and methods that carry out time-based content analysis in real time. Content analysis may involve indexing time-based content (or streams) using various audio and/or optical recognition techniques. In addition, content analysis may involve automatically and/or manually tagging content for interactions and experiences.